The disclosure generally relates to data migration from a source system to a destination system. Specifically, the disclosure relates to invoking a migration profile multiple times to migrate data items in a data directory in multiple passes.
Data migration typically involves migrating data from a source system to a destination system. In an ideal scenario, data on the source system is migrated to the destination system in one pass without errors. However, in practice, such an ideal scenario does not often happen. Errors typically occur during data migration, which will require data be migrated correctly again from the source system to the destination system. For example, when migrating an email message with multiple attachments, errors can occur when not all of the attachments are migrated correctly in an initial pass. In that case, the email message and its attachments need to be migrated again to the destination system in a second pass. In some situations, the data migration may take place over a period of time so that one or more passes subsequent to the initial pass are needed to migrate new data or data changes between the individual passes.
Conventional data migration engines that perform data migration do not typically have built-in state to keep track of migration state of individual data items during multiple passes of the data migration. Since no state information are kept, those engines typically re-migrate all the data items in subsequent pass(es) to the initial pass. This can be time consuming and unnecessary when most data items are already correctly migrated to from the source system to the destination system correctly in the initial pass.
As an improvement, some migration systems with stateless engines are integrated with a central database to which all the components of the system connect to access and write data (e.g. the minimal amount of data required to retain enough state to accomplish the migration engine's goals, such as syncing content on multiple passes). However, it is often difficult and time-consuming to build and maintain a central database that could scale up and down with the elastic nature of such migration systems. And if the central database crashes, then the entire migration system may crash. This central point of failure may create unacceptable risk for many data migrations, which are often critical to businesses to be completed quickly and efficiently.
Embodiments address these and other problems, individually and collectively, with an alternative to using a central database for persistence.